Systems and methods for user indication recognition

ABSTRACT

Various methods and apparatuses are provided to determine an occurrence of a first user indication and determine a plurality of Points of Interest (POI) corresponding to the first user indication, and to determine an occurrence of a second user indication and responsively determine a narrowed POI from the plurality of POIs. An action corresponding to the narrowed POI and the first and/or second user indication is determined and effected.

FIELD OF THE INVENTION

The present disclosure generally relates to methods and systems fordetermining points of interest in response to user indications.

BACKGROUND

Human Machine Interface (HMI) systems are presently understood assystems that allow users to interface with a machine (such as computingdevices, mobile devices, televisions, automobiles, home automationsystems, and so forth) through various human interface methods,including audible indications and physical gesture indications from auser. An HMI system may include a user indication recognition systemthat may employ any number of devices and methods to sense and determinean indication output by a user. For example, such a system may utilizemicrophones, cameras, motion detectors, and other devices to captureverbal indications spoken by a user and/or indication gestures performedby a user. These systems may then process such captured data todetermine one or more user-intended commands or inquiries and one ormore appropriate responses or actions.

Such systems are presently employed within limited environments so as toprovide a workable and limited set of commands and data that an HMIsystem may respond to or search through. For example, it is understoodthat HMI systems may be used in conjunction with an automobile to allowa user of an automobile to interface with various systems within theautomobile (for example, the radio, the climate control system, and soforth). As another example, present HMI systems may be used inconjunction with televisions to allow a user to interface with thetelevision.

Though suitable for at least some purposes, such approaches do notnecessarily meet all needs of all application settings and/or all users.For example, present HMI systems do not allow a user to interface with asingle HMI system to provide commands or inquiries regarding points ofinterest that may be within different or multiple environments.

SUMMARY

In one embodiment, an apparatus is provided including at least onesensor device configured to sense at least one user indication, the userindication being an indication gesture and/or an audible indication. Theapparatus may further include at least one processing devicecommunicatively coupled to the at least one sensor device. Theprocessing device can be configured to determine an occurrence of afirst user indication; determine a plurality of Point of Interest (POI)corresponding to the first user indication; determine an occurrence of asecond user indication; determine a narrowed POI from the plurality ofPOIs; determine an action corresponding the narrowed POI and the firstand/or second user indications; and effect performance of the action.

In another embodiment, a method is provided including sensing anoccurrence of a first user indication and determining a plurality ofPOIs corresponding to the first user indication. The method furtherincludes sensing an occurrence of a second user indication anddetermining a narrowed POI from the plurality of POIs in response tosensing the occurrence of the second user indication. The method alsoincludes determining an action corresponding to the narrowed POI and thefirst and/or second user indications; and performing the action.

In another embodiment, a method is provided including sensing anoccurrence of a first user indication and determining a plurality ofPOIs corresponding to the first user indication. The method furtherincludes sensing an occurrence of a second user indication anddetermining a narrowed POI from the plurality of POIs in response tosensing the occurrence of the second user indication. The method alsoincludes performing an action corresponding to the narrowed POI.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating functional components of a userindication recognition system in accordance with various embodiments;

FIG. 2 is a contextual example of an application setting utilizing theuser indication recognition system in accordance with variousembodiments;

FIG. 3 is a flowchart depicting a method corresponding to the userindication recognition system in accordance with various embodiments;

FIG. 4 is a flowchart depicting additional method steps corresponding tothe user indication recognition system in accordance with variousembodiments;

FIGS. 5A-5D are illustrations showing various features of the userindication recognition system in accordance with various embodiments;and

FIG. 6 is a flowchart depicting yet an additional method correspondingto the user indication recognition system in accordance with variousembodiments.

DETAILED DESCRIPTION

Before any embodiments of the disclosure are explained in detail, it isto be understood that the disclosure is not limited in its applicationto the details of construction and the arrangement of components setforth in the following description or illustrated in the followingdrawings. The disclosure is capable of other embodiments and of beingpracticed or of being carried out in various ways. Also, it is to beunderstood that the phraseology and terminology used herein is for thepurpose of description and should not be regarded as limiting. The useof “including,” “comprising,” or “having” and variations thereof hereinis meant to encompass the items listed thereafter and equivalentsthereof as well as additional items.

Additionally, as this disclosure is directed to a user indicationrecognition system 100 and method, it should be recognized that“indication” can designate any type of potential input, including, butnot limited to indication gestures including hand movements, headmovements, eye movements, body movements, or various other bio-metricobservations of a user or other actions or activities undertaken by auser, whether intentional or unintentional. Further, “indication” caninclude audible indications, including but not limited to spoken wordsand phrases, spoken utterances, humming, sung or hummed tunes,whistling, respiratory sounds, guttural sounds, clicking, clapping,snapping, and other verbal and non-verbal sounds.

Referring now to the figures, FIG. 1 illustrates an exampleconfiguration of a user indication recognition system 100 according toone approach. The system 100 includes at least one processing device102. Although depicted and often described herein as a singularprocessing device 102 (e.g., “the processing device 102”), the at leastone processing device 102 may comprise multiple distinct processingdevices 102 that may be co-located within a singular physical system,may be co-located within a singular larger architecture (such as withina larger system beyond the user indication recognition system 100), ormay be relatively co-located though part of distinct systems (forexample, a vehicle coupled to a mobile device that is within or near thevehicle). Alternatively still, the at least one processing device 102may be multiple processing devices that are physically remote from eachother. For example, various processing devices may collaborate in acloud-computing setting or in a client/server application settingwherein various portions of the required processing operations areperformed by distinct and remote processing devices that are coupledthrough known telecommunication systems and methods. Further, the atleast one processing device 102 may include any known processing devicesincluding but not limited to any of the following: a central processingunit (CPU), a microprocessor, a PIC processor, a Field-Programmable GateArray (FPGA) device, a Digital Signal Processor (DSP), a computer, aserver, a cloud-computing framework, and other devices capable ofprocessing commands and data.

The processor 102 can have multiple I/O ports to accept information frommultiple systems. The processor 102 can also be configured to implementvarious methods that are discussed in more detail below. The processor102 can also be configured to implement other processes as are necessaryfor the operation of the herein described system and method. The userindication recognition system 100 can include multiple sensor devices.Non-limiting examples of possible sensing devices include visual sensors104. Visual sensors 104 can be one or more cameras, includingcharge-coupled devices (“CCD”) or complementary metal oxidesemiconductor (“CMOS”)-based devices, stereoscopic sensors, infrared(IR) type sensors (which may be used for example to sense images in adark environment) that may be mounted about an interior of a vehicle.For example, the visual sensors 104 may be placed near a rear-viewmirror in a typical vehicle. In various embodiments, the visual sensors104 provide suitable resolution to capture and allow determination of,for example, a direction in which a user's eyes are directed, headmovements, and even fine and/or quick finger movements. In one approach,the user indication recognition system 100 can contain or be coupled tomultiple types of visual sensors 104 that can be physically mounted,placed, or otherwise located such that they can capture physicalindication gestures of a user. For example the visual sensors 104 may beplaced within the interior of a vehicle, on the front of a televisiondisplay or computer screen, or within a room to capture indicationgestures by a user within the viewing angle of the visual sensors 104.In one form, multiple visual sensors 104 can be positioned to observeseparate portions or areas of an environment that users may or may notbe occupying (such as, for example, various occupied and non-occupiedportions of the passenger compartment of a vehicle). The visual sensors104 can also be position such that they have a field of view that islimited to or includes an outer environment 206 that is outside of aninner environment 204 (see FIG. 2). By way of non-limiting example,multiple visual sensors 104 can be coupled to a vehicle to provide afield of view outside of the vehicle (the outer environment in thisexample) to provide a 360 degree viewing angle of the outer environment,or a partial viewing angle less than 360 degrees. The visual sensors 104may be used to detect visual objects both in the inner environment andin the outer environment. As described, the visual sensors 104 may bepart of a “machine vision” system that may also include the at least oneprocessing device 102. Typically, the machine vision system will includeat least two visual sensors 104 to be able to capture stereoscopicimages. With these stereoscopic images, the processing device 102 canimplement various algorithms to determine the contents of one or morecaptured images and be able to relate the determined images to anaction, function, or feature. For example, the machine vision system maybe able to capture hand gestures, finger movements, the direction inwhich a user is pointing or looking, or other macro- or micro-detailspertaining to the user.

The user indication recognition system 100 may also include or becoupled to one or more audio sensors 106. The audio sensors 106 can bemono or stereo microphones. The audio sensors 106 can also employ or beused in conjunction with noise cancellation or active noise reduction(ANR) technology to filter ambient noise (such as, for example, road andengine noise in a vehicular setting or multi-media audio output in amulti-media setting). In one approach, the audio sensors 106 can belocated within the inner environment (for example, inside the passengercompartment of a vehicle) and can be configured to capture an audioindication of a user within the inner environment. Alternatively, theaudio sensors 106 can also be located within an outer environment (forexample, outside of the passenger compartment of a vehicle). Asdescribed, the audio sensors 106 may be part of a voice recognitionsystem that may also include the at least one processing device 102.Typically, the voice recognition system will be capable of recognizingspoken words and phrases, spoken utterances, humming, sung or hummedtunes, whistling, respiratory sounds, guttural sounds, clicking,clapping, snapping, and other verbal and non-verbal sounds.

The user indication recognition system 100 may also include or becoupled to one or more telemetry sensors 108. Non-limiting examples oftelemetry sensors 108 can include weather sensors, traffic conditionsensors (in a vehicular setting), accelerometers, compasses, or othersensors configured to measure a configuration, attitude, position,speed, velocity, orientation, steering wheel angle (in a vehicularsetting), and the like. In various approaches, a given applicationsetting may already include multiple telemetry sensors of varying typesas may be required in that particular setting, which sensors may be madeavailable to the user indication recognition system 100 as needed orrequested. Additionally, the system 100 may also include or be coupledto one or more environmental sensors 110. Non-limiting examples ofenvironmental sensors 110 can include temperature sensors, humiditysensors, illumination sensors, wind sensors, or other environmentalsensors 110 configured to measure one or more environmental conditionseither outside or inside the vehicle.

The user indication recognition system 100 may also include or becoupled to one or more supplementary sensors 112 as needed. Non-limitingexamples of supplementary sensors can include pressure sensors,biometric sensors, voltage sensors, current sensors, capacitive touchsensors, etc. The supplementary sensors 112 can be used to supplementthe other sensors of the user indication recognition system 100 byproviding more detailed information where needed. It should be notedthat the functions and/or physical aspects of any of the sensorsdescribed herein, such as the visual sensors 104, the audio sensors 106,the telemetry sensors 108, the environmental sensors 110, and thesupplementary sensors 112, although categorized in this description assuch, may be emulated, reproduced, duplicated, combined with othersensors, or categorized in a different manner than described hereinwhile still providing the same or similarly suitable functionality.

The processor 102 of the user indication recognition system 100 may alsoinclude or be coupled to one or more communication modules 114. Thecommunication module 114 can be used to interface the user indicationrecognition system 100 with a mobile device (such as a cellular phone)or other mobile communication device. The communication module 114 canalso be used to communicate with other external devices or networks. Inone embodiment, the communication module 114 can include a Bluetooth®communication system for communicating with cellular phones or otherportable electronic devices. The communication module 114 can alsocommunicate with external cellular networks such as those operatingusing LTE, 4G, 3G or other cellular communication protocols as are knownin the art. The communication module 114 can also be equipped to operateusing a Wi-Fi connection such as IEEE 802.11-type networks.Additionally, the communication module 114 can also be capable ofreceiving information from Global Positioning Satellites (“GPS”).Positional information can be received via the GPS by the communicationmodule 114. It should be noted that many application settings for theuser indication recognition system 100 may already include acommunication module 114, the use of which may be made available to theuser indication recognition system 100 as needed or requested.

The processor 102 of the user indication recognition system 100 can alsobe coupled to a larger system which the user indication recognitionsystem 100 may be used with. Such coupling may be achieved via one ormore system interface modules 116. The system interface module 116 canallow the user indication recognition system 100 to have access toinformation pertaining to the larger system. For example, if the userindication recognition system 100 is installed into a vehicle,information such as speed, rpm, engine temp, GPS data, Bluetooth®communication data, audio input data, etc., can be provided to the userindication recognition system 100. Furthermore, the system interfacemodule 116 can allow for communication between some or all modules andsub-systems of the larger system and the user indication recognitionsystem 100. Additionally, via the system interface module 116, the userindication recognition system 100 may be able control the other modulesand sub-systems of the larger system. The system interface module 116can communicate with the larger system via an industry standardcommunication protocol such as CAN, CAN Open, USB, serial communication,parallel communication, or any other communication protocol as is knownin the art. Additionally, the system interface module 116 can detect thepresence of a system output 118. The system output 118 may include theoutputs of one or more modules or sub-systems that may indicate acurrent status of the larger system, as well as an indication of whetherany user interfaces of the larger system have been activated

As one contextual example, the user indication recognition system 100may be coupled to a vehicle (the vehicle being the larger system). Inthis contextual example, the system interface module 116 can providecommunication with an infotainment system, a navigation system, acompass, a climate control system, power windows, power locks, internallighting, or other system as is known in the art. Further, the userindication recognition system 100 may be able to control theinfotainment, climate control, navigation, and other systems. In someapproaches, the user indication recognition system 100 can be limited toonly controlling those vehicle systems that could not affect the safetyof operating the vehicle, but may be extended to safety features aswell. As stated above, the system interface module 116 may be able todetect a current status of the larger system, for example, a currentstatus of the power window (e.g., 50%, fully closed, currently opening,currently closing) and an indication of whether the power window buttonis current activated (and in what direction). Similarly, in the case ofa reading light, the system output may indicate a current status of thelight (e.g., on, off, 50% brightness) and an indication of whether thereading light button has been pressed. These concepts apply as easily toother settings, such as a multi-media system, whereas the userindication recognition system 100 may be able to detect a currentchannel, volume setting, etc, and whether any settings are currentlybeing or have recently been altered.

The processor 102 of the user indication recognition system 100 can alsoinclude or be coupled to a memory device 120. The memory device 120 canstore information related to features, historical data, or other datarequired to operate as described herein.

Turning now to FIG. 2, a contextual example of an application of a userindication recognition system 100 is illustrated. A user 202 is shownwithin an inner environment 204. Outside of the inner environment 204 isan outer environment 206. A boundary 208 may separate the inner 204 andouter environments 206. The boundary 208 may be a physical boundary thatphysically separates the inner 204 and outer environments 206, or may bea virtual or inferred boundary. For example, in a vehicular example (asis depicted in FIG. 2), the interior cabin of the vehicle may be theinner environment 204, whereas the areas exterior to the cabin of thevehicle may comprise the outer environment 206. By some approaches,areas that are part of the vehicle, but are not part of the passengercabin (for example, the engine area or the trunk) may be included aseither inner 204 or outer environment 206 dependent upon the needs orpreferences associated with the specific application. Typically, thoughnot necessarily, portions of the outer environment 206 will be further(e.g., physical distance) from the origination point of a userindication (e.g., the location of the user 202) than portions of theinner environment 204.

As mentioned above, the contextual example shown in FIG. 2 is avehicular application of a user indication recognition system 100.Although specifically shown here in a vehicular application, theteachings described herein may apply easily in other settings, such as,for example, other vehicle types including a motorcycle, a golf-cart orother open-air vehicle, an aircraft, a train, a boat, a submarine, aspace shuttle, or the like. Further, these teachings may also apply toother non-vehicular applications, such as a multimedia setting (e.g., ahome theater, a computer, etc.), a home automation system, an individualroom, or other settings. For example, in a multi-media setting, an innerenvironment 204 may include aspects that affect or relate directly to atelevision or multi-media system specifically (such as channel, volume,etc.), while the outer environment 206 may be related to points ofinterest external from the multi-media system (for example, a lightingdevice, the remainder of the room in which the multi-media systemresides, a neighboring room, the outdoors, etc.). By another example, ina room setting, an inner environment 204 may be the interior of the room(possibly with closable doors and/or windows, or with passageways thatare always open such as hallways or walkways that may separate, forexample, a family room from a kitchen). In such an application, an outerenvironment 206 may be areas outside of the room. For example, if theinner environment 204 is a family room, an outer environment 206 mayinclude a neighboring kitchen, dining room, bathroom, bedroom, or theoutdoors.

Further still, these teachings may apply to virtual, emulated, orreproduced settings, such as, for example, a video or virtual realitydepiction of any of the above described settings. For example, a vehicleinterior, aircraft cockpit, or a room depicted in a video game mayconstitute an inner environment 204 while areas depicted outside ofthose settings may constitute the outer environment 206. By anotherexample, a remote control station controlling an actual vehicle ordevice (for example, a drone vehicle or aircraft, a robotic device, asubmarine, a bomb-disposal robot, or other remote-controllable devices)may provide a similar application as the video game applicationdescribed above, though one or both of the environments may correspondto actual physical environments rather than virtually createdenvironments. In such an application, the real environment may bephysically remote from a user (such as a user of a remotely controlleddrone viewing a remote, yet actual physical environment in which thedrone is driving or flying). By one example, an inner environment 204for a user of a remote control station may be a virtual or emulatedenvironment (be it displayed visually or not) used to control the remotedevice (for example, by using hand gestures), whereas the outerenvironment 206 may be the reproduced images provided by the remotedevice to the user at the remote control station. In such anapplication, although the user 202 is physically remote from the pointof reference for the outer environment 206, that point of reference isestablished virtually via the interaction between the remote controlstation and the remotely controlled device.

Certain boundaries 208 may be virtual. For example, continuing with thecontextual vehicular example of FIG. 2, if a window is rolled down or aconvertible top is folded down, then a physical boundary is no longerpresent to delineate the inner 204 or outer environments 206. However,in such an instance, a boundary 208 can be inferred as the locationwhere the window or convertible top would normally reside. Similarly, inopen-air applications (such as motorcycles, golf carts, go carts, ATVs,boats, jet skis, cable cars, or any other vehicle or non-vehicleapplication that can be open-air), a boundary 208 separating inner 204or outer environments 206 may be inferred or virtually created based onthe needs and conventional wisdom associated with each specificapplication. For example, with a motorcycle, the inner environment 204may include only areas required to interface with and/or control themotorcycle and its systems directly, whereas all other areas mayconstitute the outer environment 206. In such an application, theboundary 208 will be set accordingly despite the lack of a physicalboundary.

Further still, by some approaches, the boundary 208 between the inner204 or outer environments 206 may not correspond directly or entirely toa physical boundary and may instead correspond to a functionaldelineation. For example, with respect to the motorcycle example above,the inner environment 204 may correspond only to user indicationscorrelating to functional interactions with the motorcycle and itssystems (such as turn on a blinker, turn a radio down, etc.), where asall other user indications will be considered to be relating to pointsof interest within the outer environment 206. By other approaches aninner environment 204 may be defined as items or functions that may becontinuously available to a user 202 or are in a steady-state (forexample, the systems and features relating specifically to a vehicle)where as the outer environment 206 may be defined as points of interestthat are continuously evolving or are evolvable.

With continued reference to FIG. 2, various Points of Interest (POI) maybe present in any given application setting. For example, a POI 210 maybe within the inner environment 204 and may relate to a system of avehicle (such as the infotainment center, navigation, climate control,etc.) with which a user 202 may wish to control, inquire about, orinterface with. Additionally, other POIs 212, 214 may exist in the outerenvironment 206. These may correspond to locations (such as addresses,buildings, parks, rest areas, etc.), businesses (such as stores, gasstations, retail shops, restaurants, hotels, etc.), roads or streets,objects, or any point of interest 212, 214 about which a user may wishto inquire or provide a command.

Turning now to FIG. 3, a flow chart illustrating a method 300 forrecognizing user indications is provided according to at least oneapproach, which method the at least one processing device 102, sensingdevices, or other disclosed devices may be configured to perform inpart. At step 302, a sensor device senses at least one first userindication. The first user indication can include either or both of anindication gesture (for example, a finger point) or an audibleindication (for example, a spoken command). At step 304, the processingdevice that is communicatively coupled to the sensor device determinesat least one occurrence of the first user indication. As stated above,such determination can be achieved through various algorithms andprocesses as are known in the art that are designed to determine theoccurrence of such user indications. For example, the processing device102 may be configured to analyze an orientation of a finger in3-dimensional space (for example, it's plane, angle, direction, etc.).

At step 306, the processing device 102 may determine a plurality of POIscorresponding to the first user indication. For example, the processingdevice 102 may determine a plurality of POIs based simply on a generaldirection in which an indication gesture was made, or based on criteriaof an audible indication.

By one approach, the processing device may determine whether the userindication corresponds to at least one POI within the inner 204 or outerenvironment 206 (e.g., step 402 in FIG. 4, discussed below). Forexample, the processing device 102 may determine or identify a pluralityof POIs 210 within the inner environment 204 when the processing device102 determines the user indication corresponds to at least one POIwithin the inner environment 204. Conversely, the processing device 102may determine or identify a plurality of POIs 212, 214 within the outerenvironment 206 when the processing device 102 determines the userindication corresponds to at least one POI within the outer environment206. At this point, further disambiguation may be required to determineone or more narrowed POIs to which the user was indicating.

Optionally at step 308, in one approach, (possibly if the processingdevice 102 has determined that the user is referring to a POI in theouter environment 206) the processing device 102 may take into accountother data to make a determination as to the plurality of POIs (or to aparticular POI) to which the user is referring. For example, by oneapproach, the processing device 102 may use, at least in part, locationdata (for example, GPS data), compass data (for example, to determine avehicular orientation), steering wheel angle data (for example, todetermine a current directional vector in which the vehicle is travelingor has traveled), speed data (for example, to determine a POI which theuser may have recently passed), CAN bus data (for example, to determineother relevant vehicular or system settings that may be pertinent to thePOI determination), a time of day, a direction of travel, or any otherrelevant data that may be pertinent to the plurality or a particular POIdetermination.

At step 310, the processing device may determine an occurrence of atleast one second user indication in response to determining theplurality of POIs. In at least one example, the first user indicationmay be an indication gesture (along or with other indications), whereasthe second user indication may be an audible indication, however theopposite may be true in other examples. In one approach, the furtheruser indication may be a fine user indication whereas the initial userindication may have been a coarse user indication. For example, a coarseuser indication may be a finger point, a hand directional indication, ora head nod. However, a fine user indication may constitute much finermovement such as moving a finger up or down slightly (possibly whilekeeping the arm steady). Such a finer action may, for example,correspond to navigating a menu or scrolling through a list of aplurality of possible POIs. This coarse-to-fine user indication approachis congruent with a process of culling a plurality of possible POIs downto a single or few narrowed POIs and is discussed through thedisclosure.

At step 312, the processing device 102 can determine one or morenarrowed POIs from the plurality of POIs (possible using thecoarse-to-fine user indication approach described above, or with otherapproaches, such as question and answer). At step 314, the processingdevice 102 can determine at least one action corresponding to the atleast one narrowed POI and the first and/or second user indication.

For example, in a vehicular setting, a user may provide a coarseindication by pointing or gesturing in a general direction. Theprocessing device 102 may then effect the display of a plurality of POIsthat correlate to the general direction to which the user was pointing(within the inner environment 204, within the outer environment 206, orwithin both) on a display such as the infotainment center. Suchdisplaying may include selecting and/or highlighting a set of POIs thatmay or may not already be present on the display, and may furtherinclude removing or excluding from the display POIs that are not in thegeneral direction indicated. Then, in a subsequent step, a user mayperform finer indication gestures (such as flicking a finger) to choosefrom the plurality of POIs to select one or a few narrowed POIs. Invarious approaches, the determination of a particular POI may utilize amulti-channel data approach that may include an accurate depth mappingscheme, color recognition (for skin tone, gloves, etc.), head tracking,eye tracking, and voice recognition. The combination of these variousfactors results in a more robust system.

Either or both of the plurality of POIs or the at least one narrowed POImay be determined (in steps 306 or 312) using environmental data, a timeof day, a present location, a known location, information about one ormore present users, a user setting, a historical record of POIs, orother factors that may be relevant to the selection of the plurality ofPOIs. For example, in a search setting, a user may make an audibleindication stating “show me places to eat.” Such a request for POIs mayresult in a larger number of POIs being returned in a set of searchresults. The results may include restaurants, grocery stores, coffeeshops, and the like. In response, the system may analyze the pluralityof POIs that were returned by the initial search to identify at leastone narrowed POI, thereby facilitating the user's search. For example,the system may make a determination that the current time is closer todinner time rather than lunch time, and may make a further determinationthat, based at least in part upon the time, that the user wishes to viewa listing of sit-down restaurants as opposed to fast-food locations(whereas the user may be more inclined to visit a fast-food locationduring breakfast or lunch hours) or coffee shops. The system can thendetermine the plurality of POIs (restaurants in this example) or thenarrowed set of POIs by selecting only sit-down restaurants andproviding only those POIs in a result set to the user. Similarly, thesystem may make a determination as to whether children are presentwithin the vehicle (e.g., by detecting the presences of smallerindividuals, the presence of car seats or booster seats, or the merepresence of occupants in the rear seats), and may determine that therequesting user only wishes to see fast-food locations orfamily-friendly restaurants, in which case only such POIs will bereturned in response to the user's search. These determinations may bemade automatically by the system, in which case a filtered set of POIswill be returned to the user. Or, alternatively, the system may indicateto the user that the requested search has identified a large number ofPOIs and can prompt the user to filter the POIs to identify a narrowedset of POIs. For example, the system may inform the user that the timeis close to dinner time and may inquire as to whether the user wouldwant to filter the results in a search for “places to eat” to onlyrestaurants suitable for dinner.

Similarly still, the system may review a historical database (possiblystored in the memory 120) that may provide additional information thatmay be used (possibly in conjunction with other information) to make thedetermination of the plurality of POIs or the narrowed POI. For example,the system may recognize a trend in the historical database that whenthe user executes a particular search that generates a large number ofcandidate POIs, the user tends to ultimately select one particular typeof POI. For example, the historical database may store informationdescribing prior searches for restaurants. That historical data mayindicate that the user, when executing a search for “places to eat”,almost always selects a fast-food POI instead of a sit-down restaurant.Similarly, the system may determine from the historical database thatwhen a user makes such a request and is over 20 miles from home, theuser always selects a fast-food option, but when within 20 miles fromhome, almost always selects a sit-down restaurant. After analyzing thedata stored within the historical data to make a determination as towhether the results of a particular search can be filtered to generate anarrowed set of search results in this manner, the system mayautomatically filter the results based upon that prior behavior, or canprompt the user to filter the results in accordance with their priorbehavior. Once prompted, the user can then select to filter the resultsor to view the full set of results including all POIs that match thesearch request.

Accordingly, the system can use any of these factors in generatingvarious interactive prompts, questions, or options output to a user toaid in selecting one or more narrowed POIs. For example, after aplurality of POIs are identified, the system may ask a user if they arelooking for sit-down restaurants because the system knows that when itis closer to dinner time, the user may be searching only for sit-downrestaurants as opposed to fast-food restaurants. Further still, thesystem may order or weight a plurality of POIs in a similar manner toaid the user in making further selections or indications to select thenarrowed POI (e.g., placing sit-down restaurants at the top or first ina list as the system understands the user's general preference forsit-down restaurants).

In another example, the system may alter how a plurality of POIs areselected and/or presented to a specific user making the request orcommand based on the user. Such determinations may be based on a user'scurrent role (e.g., driver or passenger), age, physical location withinan environment, disability, or other factors. For example, if a driverinquires about a place to eat, the system may provide very aggressivelimitations on the number of POIs presented in a plurality of POIs ascompared to if a passenger makes a request. This may be to limit thepresented results to a lower number to prevent distracting a driver fromthe road (visually or mentally) for too long. Conversely, if a passengermakes the request, a larger pool of POIs may be presented as theirsafety will not suffer for reviewing a larger plurality of POIs.

By another example, a user may point out a window toward a garage doorand state “open that.” The system may determine that the user isindicating either to open the window, unlock the door, or open thegarage door (e.g., a plurality of POIs) and may present that pluralityof options to the user. The system may prompt the user to make aselection between the three options to choose the narrowed POI (thegarage door in this instance). However, the system may know based on,for example, the historical database that when a current location of avehicle is within 100 feet of a home address the user almost alwayswants the garage door opened. In that case, the system may then presentthe garage door option as the first option to the user. In another form,the system may just make the final determination that the user meant toopen the garage door and perform that task rather than (or possibly inaddition to) presenting the user with the plurality of POIs.

It will be readily understood by those of skill in the art that the userindication recognition system 100 can use any number or combination offactors in determining the plurality of POIs or in determining the atleast one narrowed POI. These factors are not limited to the abovedescribed factors and methodologies, nor are they limited to being usedin the ways in which the above examples have been described. Suchfactors and methodologies may be design and/or setting specific suchthat various factors and methodologies may be employed in this regard,which variations are fully contemplated by this disclosure.

In conjunction with steps 310 and 312, in some embodiments, afterdetermining the plurality of POIs corresponding to the first userindication in step 306, the system can turn off one or more potentialPOI targets to facilitate the refinement of the plurality of POIs intothe narrowed set of POIs. For example, if, in step 306 a plurality ofPOIs are identified that are each outside of the vehicle, potential POIsthat may reside within the vehicle (e.g., POIs associated with thevehicle's entertainment system, air conditioning system, power windows,and the like) may be disabled in conjunction with step 310. This ensurethat the user's second indication (in some cases, a finer indicationgesture), which is intended to refine the set of POIs identified in step306 does not inadvertently refer to or otherwise indicate one of thePOIs inside the vehicle.

Similarly, when the first user indication determined in step 304indicates a direction towards a number of potential POIs (e.g., the userindication could be a pointing finger), the system may select andhighlight POIs in only that direction, with all other POIs beingdisabled or otherwise ignored. Having selected only POIs in thedirection indicated by the user, the user can then proceed to choosefrom the more refined set of POIs a narrowed set of POIs, perhaps basedon a voice command. The voice command could, in turn, highlight an evensmaller subset of POIs, and finally, a single POI may be chosen from anarrowed down subset with a pointing gesture, or an even finer gesture.

Returning again to FIG. 3, at step 314, the processing device 102 candetermine at least one action corresponding to the at least one narrowedPOI and the first and/or second user indication. Continuing with thevehicular contextual example used throughout, if a user pointed at awindow and made a rolling motion with their finger, the processingdevice 102 could make a determination based on both the determined POI(window) and the user indication (pointing and rolling motion) that thecorresponding action is to roll up or roll down the window. Similarly,if a user points to a restaurant at the side of the vehicle and states“remember that,” the processing device 102 can make a determinationbased on both the determined POI (restaurant) and the user indication(pointing and audible command) that a corresponding action may includesaving the location on notes, calendar, checking-in to the location, oreven making a reservation. By one approach, a voice prompt orinteractive display screen may be able to aid a user in performingfurther actions.

At step 316, the processing device 102 may effect performance of theaction determined in step 312. For example, the processing device 102may communicate with a different system through the system interface 116to effect the action (e.g., communicate with a window control module toeffect rolling up or down of the windows, or communicate with anavigation system to alter a navigation route, etc.). The processingdevice 102 may also communicate through the communication module 114 toone or more other devices to effect the action (for example, communicateto a mobile device via Bluetooth® to make a phone call, edit a calendarentry, or text a friend the user's present location). It should beunderstood that the processing device 102 itself may actually performthe action (e.g., an action to disable the user indication recognitionsystem 100), while other situations may involve the processing device102 interfacing or communicating with one or more other systems toeffect performance of the action.

Different actions can be determined and effected according to variousapproaches. In one approach, as mentioned above, a user can provide acommand to save a POI 212, 214 location. By another approach, theprocessing device 102 can determine an action to retrieve and provideadditional information about a POI. For example, a user may state “whatis that” while pointing toward, nodding toward, or looking at a POI 212,214 in an outer environment 206. The processing device 102 may thenretrieve information about the POI, possible from third-partyinformation providers (e.g., Yelp®, Zagat®, Yellow Pages®, etc.), andprovide such information (e.g., store hours, ratings, overview, etc.).By another approach, a user may indicate a POI and state “Is [friend'sname] here?” The processing device 102 may then determine whether thefriend is there (e.g., if the friend has a device that broadcasts theirpresent location) by known means, including, for example, bycommunicating with the user's mobile device to make such adetermination.

In further approaches, the processing device 102 can effect actionsrelating to a navigation system by indicating POIs in the outerenvironment 206. In one example, a user may point to another vehicle(e.g., a disabled or crashed vehicle) or other obstacle (e.g., debris orpotholes in a roadway) and identify it has a road hazard. The processingdevice 102 may then cause the navigation system to mark the hazard assuch and/or communicate the hazard to other individuals or services thatwould benefit from such information. In another example, a user mayalter a navigation route on the fly by pointing to a different streetand asking, for example, “Can I take a left here to avoid the accident?”The processing device 102 may then effect an action to, for example,compare the suggested route with traffic information that is available(possibly to a vehicle or a coupled mobile device) to determine if theproposed route is good (e.g., meets certain criteria).

In even further approaches, the processing device 102 can effect actionsrelating to controlling other devices or appliances that are nottypically coupled to the user indication recognition system 100. Forexample, a user may pull up to a movable barrier (such as a garage dooror a gate) and point to the movable barrier and state “open the door.”In response, the processing device 102 may effect operation of a movablebarrier operator (e.g., garage door opener or gate operator) to open thebarrier. The barrier operator may be communicatively linked to thesystem (e.g., the user indication recognition system 100, or a largersystem to which the system 100 is coupled). This same principle appliesto other remote appliances, such as, for example, lighting controllers,alarm system, and other home-automation systems.

In one form, the user indication recognition system 100 may be used tocontrol aspects of the larger system (e.g., a vehicle, home-theater,etc.) while the user is in the outer environment 206. For example, auser may be outside of a vehicle and point to the vehicle and state“lock it” or “unlock it.” The user indication recognition system 100 mayrecognize the speaker through various voice recognition techniques or byusing other biometric recognition or authentication techniques (e.g., anauthenticated key fob) to validate the user as a user with authority toprovide the command. A similar situation may apply in home-theatersettings if a user walks out of the room and into an outer environment206 and states, for example, “pause” or another command. The processingdevice 102 may determine to effect the action even though the user isnot physically present within the inner environment 204.

In still other approaches, the processing device 102 can effectperformance of actions relating to controlling or inquiring aboutaspects or features of the larger system to which the user indicationrecognition system 100 is coupled (e.g., a vehicle, etc.). In oneexample, and continuing with the contextual vehicular example, a usercan point at the radio and inquire “who is this?” In response, theprocessing device 102 may effect performance of an action that shows thecurrent song title, artist, etc. on a display screen, and/or possiblyprovides an audible version of the same information. Other examplesinclude pointing at a navigation screen when a traffic warning appearsto cause the system to read the traffic warning out loud, or pointing toa vehicle warning light when it appears to provide the user withadditional information as to what the warning means and how to remedyit.

By further approaches, the processing device 102 can effect performanceof actions relating to inquiring about individual contacts or friends.In one example, and continuing with the contextual vehicular example, auser can point at the navigation system and say “Show my [contact'sname].” The processing device 102 can then effect performance by thenavigation system to, for example, bring up the contact locationinformation. In response, a user can initiate further actions throughthe user indication recognition system 100 or through the navigationsystem directly. Such possible further actions may include but are notlimited to calling or sending a message to the contact or sending thecontact's current location with live updates or the contact's savedlocation to the user's or the contact's friends.

Turning now to FIG. 4, optional additional method steps 400 aredescribed in accordance with various approaches. Such method 400 may beperformed in conjunction with the method 300 of FIG. 3, and may also beperformed, at least in part, by the processing device 102. These methodsteps may be performed in conjunction with each other or independent ofeach other.

At optional step 402, the processing device 102 may determine whether auser indication (such as the first user indication) corresponds to atleast one POI within an inner environment 204 or at least one POI withinan outer environment 206. In one form, this step may be performed aspart of step 306 of the method 300 of FIG. 3. For example, with briefreference to FIG. 2, the processing device 102 may determine whether theuser 202 is referring to a POI 210 within the vehicle (e.g., innerenvironment 204), or one or more POIs 212, 214 outside the vehicle(e.g., outer environment 206).

By one approach, a determination of whether the user indicationcorresponds to a POI within the inner 204 or outer environment 206 canbe made by determining a known relationship between the user indicationand a specific POI. For example, if a user points directly to theinfotainment center of a vehicle, as is illustrated in FIGS. 5C and 5D,or gives a spoken command relating directly to the radio or the climatecontrol, the processing device 102 can determine that the user 202 isreferring to a POI 210 within the inner environment 204. By anotherapproach, continuing with the vehicular example, if the user 202 makesan indication gesture generally aimed toward the dashboard, seats,floor, interior side panels of the doors, or toward other areas of theinterior of a vehicle that may obstruct a view of the outer environment206, the processing device 102 may be able to infer that such anindication gesture is generally relating to a POI within the innerenvironment 204. This may be because a user may not point to or gesturetoward a POI 212, 214 in the outer environment 206 that the user cannotsee. Further, from a user's perspective in 3-dimensional space, POIs maynot generally physically exist below a user's sight-line because thesight-line will typically extend downward toward the ground within a fewfeet of the vehicle (except possibly for POIs including a current roadon which the vehicle is traveling or a current location at which thevehicle is located). Typically, though not always, POIs 212, 214 in anouter environment 206 may exist on the same horizontal plane as avehicle or higher (e.g., a store on the side of the road or an office ina tall building). Thus, a user may point sideways or forward out of awindow at a POI 212, 214 located in an outer environment 206, which byone approach, the processing device 102 can use in determining whetherthe POI is within the inner 204 or outer environment 206. Such anindication gesture is illustrated in FIG. 5A. In such an approach, theprocessing device 102 may need to determine whether the user 202 isreferring to the POIs 212, 214 in the outer environment 206 or thewindow, windshield, or door as part of the inner environment 204.However, these determinations can be made by any number of methods. Forexample, the processing device 102 may take into account whether auser's eyes are also directed toward a potential POI which the user isgesturing to (as the user may not look at a window when making a gesturepertaining to the window as the user always knows where the window is,but may look at a POI 212, 214 that is in the outer environment 206 asit is catching the user's immediate attention. In another method, auser's lips may be tracked and speech associated with a given commandmay also be extracted. By tracking the lips of a user, the system isable to mitigate false positives in voice commands, while limitingcommands associated with gestures to ones that are actually beinguttered, i.e. when the users lips indicate that the user is speaking orgenerating a command.

By another approach, the processing device 102 may take into account theactual indication gesture or the content of the audible indication whendetermining if the user is referring to a POI within the inner 204 orouter environment 206. For example, a user may point at the window andmake, for example, a rolling motion with the finger tip, indicating toroll the window up or down, in which case the processing device 102 maydetermine that the POI is the window, which is within the innerenvironment 204. However, a user may point toward the window with adifferent gesture motion that may indicate a POI in the outerenvironment 206 (for example, a pointing action that typicallycommunicates “go further,” or a repeated pointing motion). As will beunderstood by those of skill in the art, any number of indicationgestures can be utilized that the processing device 102 can beconfigured to use to distinguish between POIs within the inner 204 orouter environment 206.

By another example, a user 202 may speak an inquiry or command that istypically related to a POI 210 within the inner environment 204 (forexample “turn on radio”), in response to which the processing device 102determines that the POI is within the inner environment 204. Conversely,a user may speak an inquiry that would typically refer to a POI 212, 214within the outer environment 206 (for example, “what is that?” or“remember that”). By using such an audible indication, the processingdevice 102 determines that the POI is within the outer environment 206and can proceed accordingly. In such an example (having only an audibleindication), the processing device 102 may also utilize other driverdata possibly to determine whether the audible indication refers tosomething in the inner or outer environments, or to determine thespecific POI to which the user is referring. For example, the processingdevice 102 may refer to the direction in which the user's eyes arelooking or were looking immediately before or after the audibleindication. As will be understood by those of skill in the art, anynumber of audible indications can be utilized which the processingdevice 102 can be configured to use to distinguish between POIs withinthe inner 204 or outer environment 206.

By yet another approach, the processing device 102 may receive a userindication that includes both an indication gesture and an audibleindication. In one example, a user may point to a POI and simultaneously(or within a defined window of time before or after the gesture) state acommand or inquiry. For example, and continuing with the vehicularexample setting, a user may point toward a restaurant (POI 212) as thevehicle is stopped or passing near it and state “remember that.” By thiscombination of a gesture indication and an audible indication, theprocessing device 102 will have additional data by which to make adetermination as to whether the referenced POI is in the inner 204 orouter environment 206.

The processing device 102 may determine at least one POI (or a pluralityof POIs) within the determined environment. For example, the processingdevice 102 may determine at least one POI 210 (or a plurality of POIs210) within the inner environment 204 when the processing device 102determines the user indication corresponds to at least one POI withinthe inner environment 204. Conversely, the processing device 102 willdetermine at least one POI 212, 214 (or a plurality of POIs 212, 214)within the outer environment 206 when the processing device 102determines the user indication corresponds to at least one POI withinthe outer environment 206. Such a determination may be made simply bydetermining one or more particular POIs 210 within the inner environment204 that the user is specifically addressing or that the user isspecifically gesturing toward. Further, such a determination may be madeby selecting one or more POIs 212 within an outer boundary 216(discussed further below) if the outer environment 206 is indicated.However, by one approach, if a directional indication is provided (e.g.,a finger point, hand gesture, head nod, eye movement, etc.), theprocessing device 102 may be able to select a particular one POI or aset of POIs within the outer environment 206. In certain forms, the atleast one processing device 102 compares and/or matches user indicationsto a POI database to select the one POI or a set of POIs (which POIdatabase may be stored locally, such as at a vehicle, possibly in thememory 120, or may be stored remotely, such as in the cloud or at aserver).

With continued reference to FIG. 4, at step 404, the processing device102 may effect presentation of the plurality of POIs to the user. Forexample, the processing device 102 may effect a visual representation ofthe plurality of POIs on a display device (e.g., by listing a number ofpotential POIs on a display screen) or effect an audible representationof the plurality of POIs (e.g., by using a voice synthesizer to name thepotential POIs to the user). Many known variations regarding presentingthe plurality of POIs to a user are possible, all of which are withinthe ambit of the present disclosure.

At step 406, the processing device 102 may enable a user to interactwith the presentation of the plurality of POIs. In some cases, theprocessing device 102 may question the user as to which of a number ofPOIs was referred to by the user. The user may provide a further secondindication to the processing device 102 to indicate which POI wasactually being referred to. For example, the user could speak the nameof the desired POI, hold up a number of fingers identifying the desiredPOI, or otherwise provide some form of indication back to the processingdevice 102 identifying the desired POI. Many variations are possible forinteracting with the system to provide a second user indication, all ofwhich are within the ambit of the present disclosure.

In a similar fashion to step 402, by step 408, the processing device maydetermine whether a user indication (such as the second user indication)corresponds to at least one narrowed POI within an inner environment 204or at least one narrowed POI within an outer environment 206. In oneform, this step may be performed as part of step 312 of the method 300of FIG. 3. For example, with brief reference to FIG. 2, the processingdevice 102 may determine whether the user 202 is referring to a narrowedPOI 210 within the vehicle (e.g., inner environment 204), or one or morePOIs 212, 214 outside the vehicle (e.g., outer environment 206).

In various embodiments of the present system, searches for candidatePOIs will occur within a target search region. Because, in the majorityof instances, the POI search will be performed in response to apositional indication made by the user (e.g., by pointing a fingertowards a region in which a particular POI may be located), a defaulttarget search region may be defined that is in relative proximity to theuser.

The default search region may be defined as having any suitable shape(e.g., circle, ellipsis, square, rectangle, sphere, cube, and the like)to encompass a desired 2-dimensional or 3-dimensional space. The searchregion may encompass the user or may not encompass the user. FIG. 2, forexample, shows an example default search region having outer border 216.The default search region is defined about the inner environment 204(here, being the interior of a vehicle) and enables an operator orpassenger 202 within vehicle to search for POIs in proximity to thevehicle.

The defined outer border 216 of the default target search region mayvary based upon the application of the present system. Where the user202 resides within a vehicle, the outer border 216 of the default targetsearch region may be a circle having a radius of 100 meters from theuser (e.g., to include POIs that the user may be able to see from thevehicle). Similarly, where the system is being used in a home or officebuilding, however, the outer border 216 of the default search region maybe a circle having a radius of 10 meters away from the user, again toinclude POIs that the user can see. The default search regions can bedefined by the user or can be system-defined. The optimal default searchregions can be selected based upon experimental data (e.g., by analyzingreal-world POI searches and the result location of correctly-identifiedPOIs in response to those searches).

Even though a default search region may be optimized, there can be manycircumstances that, depending upon the user's current activity orcurrent location, can invalidate or make non-optimal the default searchregion. For example, where the present system is utilized within avehicle, a default search region that is well-optimized for city drivingwhere candidate POIs are densely populated may be entirely inadequate ina region that is sparsely populated and where candidate POIs are spacedapart. Where the default search region is optimized for city driving, aPOI search executed in a sparsely populated region may return too fewPOIs to make for an effective search. Similarly, a default search regionthat is optimized for a sparsely populated region may return far toomany POIs for a search executed within a densely populated citylocation.

Accordingly, in an embodiment of the present system, the outer border216 of a particular search region can be automatically adjusted for agiven POI search to improve the number of results returned to the user.For example, when the user and the system are in a vehicle located in anarea having a high density of the searched item(s), i.e. POIs 212, 214,the system may keep its default search radius (which could, for example,be optimized for city driving). However, if the system is in an areawith a low density of the searched items (e.g., a rural area), thesystem can increase the search radius until the system can find adesired number of results. The system can do this by implementing analgorithm to determine when to increase the radius for a particular setof search results.

FIG. 6, for example, is a flow chart illustrating a method foroptimizing an outer border of a search region for a given POI search. Instep 1000, the system receives a search input. The search input, asdescribed herein, may comprise any suitable input provided by a user tosearch for a particular POI. The search input may include an indicationgesture (such as a pointed finger) and/or additional input (such as aaudible indication, for example, a search command). The search inputenables the present system to search for POIs that match the searchinput.

Having received the search input, in step 1002 the system may define adefault search outer border 216. The default search outer border 216 maybe defined by user settings or specified as part of a system setting. Insome cases, different default outer borders 216 may be defined fordifferent types of searches. For example, if the search is for a placeto eat (e.g., a search for a restaurant), the outer border 216 may havea different size and/or shape than if the search was for a gas station.The default search outer border 216 may be affected by user preferences.For example, the user may specify that the outer border 216 should belarger if the user profile indicates that the driver prefers freewaysover local streets. Other factors affecting the default outer border 216may include a user profile, a current location, a type of search (e.g.,restaurants vs. gas stations), a threshold value (e.g., an expectationto receive between 3 and 6 results every time), a border shape, a typeof current road (i.e., city street vs. rural road), a time of day, aswell as other factors.

Having set the default search outer border 216, in step 1004 the systemexecutes the search for POIs residing within the default search outerborder 216. Having executed the search, a number of POIs are returned inresponse to the executed query. The system then determines whether thenumber of POIs returned in the search results is adequate.

If the number of results falls is greater than an upper threshold(indicating that too many POIs were returned in the search results,potentially overwhelming the user), the system, in step 1006 reduces thesize of the search outer border 216 to generate a modified outer border216 comprising a contracted outer border 216. Having reduced the size ofthe outer border 216, the search is then re-executed in step 1004 withinthe modified outer border 216. Now, with the size of the outer border216 reduced, the number of POIs that may be returned in the searchresults should decrease and be more manageable to the user.

If the number of results falls below a lower threshold (indicating thatinsufficient POIs were returned in the search results), the system, instep 1008 increases the size of the search outer border 216 to generatea modified enlarged outer border 216. Having increased the size of theouter border 216, the search is then re-executed in step 1004 within themodified outer border 216. Now, with the size of the outer border 216increased, the number of POIs that may be returned in the search resultsshould increase.

If, however, the number of results in the search results falls betweenthe lower and upper threshold, that indicates that an appropriate numberof results were returned. As such, in step 1010, the results can bedisplayed or otherwise presented to the user for review.

According to the method depicted in FIG. 6, the size of the searchregion could be iteratively adjusted—either increased or decreased anumber of times—to achieve the desired number of search results. Forexample, after the first time the size of the outer border 216 isadjusted (either increased or decreased) if the number of resultsreturned in the re-executed search does not fall between the upper andlower threshold, the size of the outer border 216 may be again adjustedand the search executed for a third time. In various embodiments,however, the number of times that the size of the outer border 216 ofthe search region is adjusted is limited. In one embodiment, the outerborder 216 may only be adjusted a single time.

In various embodiments where the system is implemented within a vehicle,the outer border 216 of a particular search region may be adjusteddepending on the vehicle's status and user's profile. For example, ifthe vehicle is driving at a high speed, the system can adjust the outerborder 216 (e.g., by increasing the size of outer border 216) becausethe car is moving quickly past candidate POIs. Conversely, as thevehicle slows down, the size of the outer border 216 may be reduced.

In one specific embodiment, the outer border 216 may be defined as anellipsis. In that case, when the system is utilized within a vehicle,the elliptical outer border 216 of the search area could be configuredso that the major axis of outer border 216 is oriented in the vehicle'sdirection of travel and has a length that is based at least somewhatupon the vehicle's speed. In that case, the minor axis of the ellipticalouter border 216 of the search region could be sized proportionally tothe POI density in the vehicle's current location. The vehicle may beplaced at a focus of the ellipse. This would result in a search areathat is skewed toward the vehicle's direction of travel, which wouldbeneficially return more POIs that are ahead of the vehicle. In someembodiments, the elliptical outer border 216 of the search region may beweighted a distance ahead of the vehicle in the vehicle's direction oftravel or along the road the vehicle is currently traveling upon so asto encompass approaching POIs. In that case, the distance that theelliptical outer border 216 is placed ahead of the vehicle may be atleast partially determined by the vehicle velocity of travel.

Accordingly, by utilizing a modified outer border 216 for a particularsearch region, the present system can control the search radius toensure that the user is presented with a reasonable number of results inresponse to a search. In an example, a car and driver are travellingfrom one major city to another, e.g., going from Los Angeles, Calif. toLas Vegas, Nev. In such an example, there may be regions of the journeywhere there are no major POIs for miles ahead of the vehicle followed byregions (e.g., within one of the cities) with a very large number ofPOIs in one relatively confined area. The present system, therefore,when miles away from the densely populated POIs, can increase the searchradius to collect information about those POIs and generate searchresults that incorporate those POIs. Conversely, the system may alsodecrease the search radius to avoid overwhelming the driver bypresenting too large a number of POIs when the vehicle is located withina region having densely populated POIs.

So configured, a user indication recognition system is provided that canallow for narrowing of selections of POIs by a user and/or on the fly bythe system. As such, a preoccupied user (such as a driver of a vehicle)may be able to quickly and precisely determine a POI with less cognitivedistractions. Also, the user indication recognition system cansimultaneously accommodate multiple distinct environments (for example,an inner 204 and outer environment 206). This can be particularlybeneficial when one environment (for example, an outer environment) isevolving or evolvable such that information may be sought or commandsgiven by a user with respect to new or evolving POIs. By utilizing thedisclosed system according the various approaches, a user can easily,quickly, and safely access information about or provide commandsrelating to POIs, possibly within the multiple environments. Forexample, this helps prevent illegal or unsafe use of a vehicle's devicesor systems (e.g., the radio or navigation) or mobile devices by anoperator while operating the vehicle. Further, various approachesdescribed herein can be provided with respect to multiple environmentswhile still maintaining usage of the user indication recognition system100 with respect to more commonly used systems and elements within aninner environment.

Various embodiments of the present invention may be embodied in manydifferent forms, including, but in no way limited to, computer programlogic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, server computer, or generalpurpose computer), programmable logic for use with a programmable logicdevice (e.g., a Field Programmable Gate Array (FPGA) or other PLD),discrete components, integrated circuitry (e.g., an Application SpecificIntegrated Circuit (ASIC)), or any other means including any combinationthereof.

Computer program logic implementing all or part of the functionalitypreviously described herein may be embodied in various forms, including,but in no way limited to, a source code form, a computer executableform, and various intermediate forms (e.g., forms generated by anassembler, compiler, linker, or locator). Source code may include aseries of computer program instructions implemented in any of variousprogramming languages (e.g., an object code, an assembly language, or ahigh-level language such as C, C++, or JAVA) for use with variousoperating systems or operating environments. The source code may defineand use various data structures and communication messages. The sourcecode may be in a computer executable form (e.g., via an interpreter), orthe source code may be converted (e.g., via a translator, assembler, orcompiler) into a computer executable form.

The computer program may be fixed in any form (e.g., source code form,computer executable form, or an intermediate form) in a tangible storagemedium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM,EEPROM, or Flash-Programmable memory), a magnetic memory device (e.g., adiskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PCcard (e.g., PCMCIA card), or other memory device. The computer programmay be distributed in any form as a removable storage medium withaccompanying printed or electronic documentation (e.g., shrink wrappedsoftware), preloaded with a computer system (e.g., on system ROM orfixed disk), or distributed from a server or electronic bulletin boardover the communication system (e.g., the Internet or World Wide Web).

Hardware logic (including programmable logic for use with a programmablelogic device) implementing all or part of the functionality previouslydescribed herein may be designed using traditional manual methods, ormay be designed, captured, simulated, or documented electronically usingvarious tools, such as Computer Aided Design (CAD), a hardwaredescription language (e.g., VHDL or AHDL), or a PLD programming language(e.g., PALASM, ABEL, or CUPL).

Programmable logic may be fixed either permanently or temporarily in atangible storage medium, such as a semiconductor memory device (e.g., aRAM, ROM, PROM, EEPROM, or Flash-Programmable memory), a magnetic memorydevice (e.g., a diskette or fixed disk), an optical memory device (e.g.,a CD-ROM), or other memory device. The programmable logic may bedistributed as a removable storage medium with accompanying printed orelectronic documentation (e.g., shrink wrapped software), preloaded witha computer system (e.g., on system ROM or fixed disk), or distributedfrom a server or electronic bulletin board over the communication system(e.g., the Internet or World Wide Web).

The present disclosure describes preferred embodiments with reference tothe Figures, in which like numbers represent the same or similarelements. Reference throughout this specification to “one embodiment,”“an embodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

The described features, structures, or characteristics of the inventionmay be combined in any suitable manner in one or more embodiments. Inthe description, numerous specific details are recited to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

The schematic flow chart diagrams included are generally set forth aslogical flow-chart diagrams. As such, the depicted order and labeledsteps are indicative of one embodiment of the presented method. Othersteps and methods may be conceived that are equivalent in function,logic, or effect to one or more steps, or portions thereof, of theillustrated method. Additionally, the format and symbols employed areprovided to explain the logical steps of the method and are understoodnot to limit the scope of the method. Although various arrow types andline types may be employed in the flow-chart diagrams, they areunderstood not to limit the scope of the corresponding method. Indeed,some arrows or other connectors may be used to indicate only the logicalflow of the method. For instance, an arrow may indicate a waiting ormonitoring period of unspecified duration between enumerated steps ofthe depicted method. Additionally, the order in which a particularmethod occurs may or may not strictly adhere to the order of thecorresponding steps shown. Some embodiments provided for are describedas computer-implemented method claims. However, one of ordinary skill inthe art would realize that the method steps may be embodied as computercode and the computer code could be placed on a tangible, non-transitorycomputer readable medium defining a computer program product.

Although the above discussion discloses various exemplary embodiments ofthe invention, it should be apparent that those skilled in the art canmake various modifications that will achieve some of the advantages ofthe invention without departing from the true scope of the invention.

What is claimed is:
 1. An apparatus, comprising: at least one sensordevice configured to sense a first user indication and a second userindication, wherein both of the user indications include an indicationgesture or an audible indication and are user actions, wherein neitherof the first user indication and the second user indication areassociated with operation of a map; a processing device communicativelycoupled to the at least one sensor device, the processing deviceconfigured to: determine an occurrence of the first user indication; inresponse to the first user indication, determine a plurality of Pointsof Interest (POIs) corresponding to the first user indication; determinean occurrence of the second user indication; determine whether thesecond user indication is associated with a POI determined in responseto the first user indication and whether the second user indication isassociated with a POI outside of the plurality of POIs determined inresponse to the first user indication; when it is determined that thesecond user indication is associated with the POIs determined inresponse to the first user indication, determining a narrowed POI fromthe plurality of POIs; turning off one or more potential POI targetsassociated with the second user indication based on the second,non-map-related operation user indication being associated with, the POIoutside of the plurality of POIs determined in response to the firstuser indication; after determining the occurrence of both the first userindication and the second indication, determine an action correspondingto the narrowed POI based on the first user indication and the seconduser indication; and effect performance of the action based on the firstuser indication and the second user indication.
 2. The apparatus ofclaim 1, wherein the first user indication is a coarse user indicationincluding an indication gesture associated with a direction.
 3. Theapparatus of claim 1, wherein the second user indication is a fine userindication associated with the audible indication.
 4. The apparatus ofclaim 1, wherein the processing device is further configured to effectpresentation of the plurality of POIs to a user in response todetermining the occurrence of both the first user indication and thesecond indication.
 5. The apparatus of claim 1, wherein the processingdevice is further configured to determine the plurality of POIs usingenvironmental data, a time of day, a present location, a known location,information associated with a present user, a user setting, or anhistorical record of POIs.
 6. The apparatus of claim 1, wherein thefirst user indication is a coarse user indication associated with theaudible indication.
 7. The apparatus of claim 1, wherein the second userindication is a fine user indication including the indication gesture.8. The apparatus of claim 1, wherein the processing device is configuredto effect performance of the action comprises the processing devicebeing configured to: enable saving a location of the narrowed POI;enable provision of information pertaining to the narrowed POI; enableprovision of information pertaining to a present location of a contactin relation to the narrowed POI; identify the narrowed POI as a hazard;enable a modification or an initiation of a navigation route; or enableoperation of a barrier operator.
 9. The apparatus of claim 1, whereinthe processing device is configured to determine the plurality of POIsby using location data, compass data, steering wheel angle data, speeddata, or controller area network (CAN) bus data.
 10. A method,comprising: sensing an occurrence of a first user indication, whereinthe first user indication is a coarse user indication; determining aplurality of Points of Interest (POIs) corresponding to the first userindication; sensing an occurrence of a second user indication, whereinthe second user indication is a fine user indication, wherein both ofthe user indications are user actions, wherein neither of the first userindication and the second user indication are associated with operationof a map; determining whether the second user indication is associatedwith a POI determined in response to the first user indication andwhether the second user indication is associated with a POI outside ofthe plurality of POIs determined in response to the first userindication; when it is determined that the second user indication isassociated with the POIs determined in response to the first userindication, determining a narrowed POI from the plurality of POIs;turning off one or more potential POI targets associated with the seconduser indication based on the second, non-map-related operation userindication being associated with the POI outside of the plurality ofPOIs determined in response to the first user indication; determining anaction corresponding to the narrowed POI based on the first userindication and the second user indication; and performing the actionbased on the first user indication and the second user indication. 11.The method of claim 10, comprising presenting the plurality of POIs to auser in response to determining the plurality of POIs using a visualrepresentation of the plurality of POIs on a display device or anaudible representation of the plurality of POIs.
 12. The method of claim11, comprising enabling the user to interact with the presentation ofthe plurality of POIs based on the second user indication.
 13. Themethod of claim 10, comprising determining a default outer border basedon a user profile, a current location, a type of search, a thresholdvalue, a border shape, a user preference for a type of road, or a timeof day.
 14. The method of claim 10, wherein the first user indicationincludes an indication gesture associated with a direction.
 15. Themethod of claim 10, wherein the second user indication includes anaudible indication.
 16. A method, comprising: sensing an occurrence of afirst user indication, wherein the first user indication is a coarseuser indication including a gesture indication; determining a pluralityof Points of Interest (POIs) corresponding to the first user indication;sensing an occurrence of a second user indication, wherein the seconduser indication is a fine user indication including an audibleindication, wherein both of the user indications are user actions,wherein neither of the first user indication and the second userindication are associated with operation of a map; determining whetherthe second user indication is associated with a POI determined inresponse to the first user indication and whether the second userindication is associated with a POI outside of the plurality of POIsdetermined in response to the first user indication; when it isdetermined that the second user indication is associated with the POIsdetermined in response to the first user indication, determining anarrowed POI from the plurality of POIs; turning off one or morepotential POI targets associated with the second user indication suchthat selection is disabled based on the second, non-map-relatedoperation user indication being associated with the POI outside of theplurality of POIs determined in response to the first user indication;determining an action corresponding to the narrowed POI based on thefirst user indication and the second user indication; and performing theaction based on the first user indication and the second userindication.